Inverse Rendering with a Morphable Model: A Multilinear Approach

In this paper, we present a complete framework to inverse render faces from single images using a 3D Morphable Model (3DMM). A 3DMM is a linear statistical model of 3D shape and texture [2]. In general, inverse rendering of faces from single photographs is ill-posed, as the same appearance can be obtained by different underlaying factors. For instance, a red pixel can be caused by skin colour, red illumination, or an increased camera sensitivity in the red channel. A combination of these factors is also possible. For an object of known shape under complex natural illumination, the well known work of Ramamoorthi [4] shows how the spherical harmonic domain can be used to estimate one or more of: illumination, surface texture and reflectance properties. We revisit this classical formulation in the context of 3DMMs. Previous methods for fitting a 3DMM based on analysis-by-synthesis recover all parameters in a single, nonconvex objective function [2, 3]. To reduce the threat of getting stuck in local minima, Romdhani introduced a fitting algorithm which incorporates features like edges and specular highlights into the cost function [5]. These fitting algorithms make limited assumptions about the illumination environment and only model ambient light and one directional light source. Zhang and Samaras [6] used spherical harmonics to model unconstrained illumination, although at the cost of assuming a simple Lambertian reflectance model.